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Abstract. The paper examines the discrete-time dynamics of neuron models (of excita- 
tory and inhibitory types) with piecewise linear activation functions, which are connected 
in a network. The properties of a pair of neurons (one excitatory and the other inhibitory) 
connected with each other, is studied in detail. Even such a simple system shows a rich 
variety of behavior, including high-period oscillations and chaos. Border-collision bifur- 
cations and multifractal fragmentation of the phase space is also observed for a range of 
parameter values. Extension of the model to a larger number of neurons is suggested un- 
der certain restrictive assumptions, which makes the resultant network dynamics effectively 
one-dimensional. Possible applications of the network for information processing are out- 
lined. These include using the network for auto-association, pattern classification, nonlinear 
function approximation and periodic sequence generation. 

Keywords: excitatory-inhibitory neural networks, chaos, nonlinear dynamics. 



1. Introduction 

Since the development of the electronic computer in the 1940s, the serial processing compu- 
tational paradigm has successfully held sway. It has developed to the point where it is now 
ubiquitous. However, there are many tasks which are yet to be successfully tackled computa- 
tionally. A case in point is the multifarious activities that the human brain performs regularly, 
including pattern recognition, associative recall, etc. which is extremely difficult, if not impos- 
sible to do using traditional computation. 

*I would like to thank Prof. S. K. Pal (MIU, ISI) for his constant encouragement. 
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This problem has led to the development of non-standard techniques to tackle situations 
at which biological information processing systems excel. One of the more successful of such 
developments aims at "reverse-engineering" the biological apparatus itself to find out why and 
how it works. The field of neural network models has grown up on the premise that the massively 
parallel distributed processing and connectionist structure observed in the brain is the key 
behind its superior performance. By implementing these features in the design of a new class 
of architectures and algorithms, it is hoped that machines will approach human-like ability in 
handling real-world situations. 

Extremely simplified models of neurons connected in a network via suitable connection 
weights were known to implement various logical functions since the 1940s. The subject re- 
ceived fresh impetus a decade and half ago due to some breakthroughs. These were however 
restricted to networks subject to convergent dynamics. Such systems tend to a time-independent 
solution (a "fixed-point" attractor) after starting off from some initial condition. On the other 
hand, the brain never settles down to a steady state but appears to exhibit a rich variety of 
non-periodic behavior. 

The development of nonlinear dynamical systems theory - in particular, the discovery of 
"deterministic chaos" in extremely simple systems - has furnished the theoretical tools necessary 
for analyzing non-convergent network dynamics. Several neurobiological studies have in fact 
indicated the presence of chaotic dynamics in the brain Q and its possible role in biological 
information processing. Thus, the ability to design networks with aperiodic behavior promises 
to add a new dimension to our understanding of how the brain works. 

Several efforts in designing and applying chaotic neural networks have already appeared Q 
- ||. (For further references see 0). In the present work, a particularly simple model whose 
state is updated after discrete time intervals is studied. The piecewise linear nature of the model 
neuron used, not only makes detailed theoretical analysis possible, but also enables an intuitive 
understanding of the dynamics, at least for a small number of connected elements. This makes it 
easier to extrapolate to larger networks and suggest possible applications. The proposed model 
is also particularly suitable for hardware implementation using operational amplifiers (owing to 
their piecewise linear characteristics). 

The rest of the paper is organized as follows. The basic features of the piecewise linear neural 
model used is described in section 2, along with the biological motivation for such a model. The 
next section is devoted to analyzing the dynamics of a pair of excitatory and inhibitory neurons, 
with self- and inter-connections. This simple system shows a wide range of behavior including 
periodic oscillations, chaos and border-collision bifurcations Q. Section 4 extends the model 
to larger networks under certain restrictive conditions. This is followed by a discussion of the 
possible application of the model to various information processing tasks, such as associative 
memory and nonlinear function approximation. The rich dynamics of the system allows it to 
respond to specific inputs with periodic or aperiodic responses (in contrast to a time-independent 
constant output, as in convergent networks) and also to act as a central pattern generator. We 
conclude with a short discussion on possible directions for further studies. 
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2. Piecewise Linear Neuron 

Let u n denote the activation state of a model neuron at the re— th time interval. If u n = 1, the 
neuron is considered to be active (firing), and if u n = 0, it is quiescent. Then, if v n is the input 
to the neuron at the nth instant, the discrete-time neural dynamics is described by the equation 

u n = F(v n ), (1) 

assuming there to be no effects of delay. The input v n is the weighted sum of the activation 
states at the (re — l)-th instant of all other neurons connected to the neuron under consideration, 
together with external stimulus (if any) . The form of J- is decided by the input-output behavior 
of the neuron. Usually, it is taken to be the Heaviside step function, i.e., 

F(z) = 1, if z > n, ,^ 
= 0, otherwise, 

where u is known as the threshold. 

If the mean firing rate, i.e., the activation state averaged over a time interval, is taken as the 
dynamical variable, then a continuous state space is available to the system. If X n be the mean 
firing rate at the re— th time interval, then 

X n+1 = F^EjWjXi + In). (3) 

Here, F is known as the activation function and re is the parameter associated with it. The first 
term of the argument represents the weighted sum of inputs from all neurons connected with 
the one under study. Wj is the weightage for the connection to the jth neuron. /„ represents 
the external stimulus at the nth instant. 

Considering the detailed biology, there are two transforms occurring at the threshold element. 
At the input end, the impulse frequency coded information is transformed into the amplitude 
modulation of the neural current. For single neurons, this pulse- wave transfer function is linear 
over a small region, with nonlinear saturation at both extremities. At the output end, the 
current amplitude is converted back to impulse frequency. The wave-pulse transfer-function for 
single neurons is zero below a threshold, then rises linearly upto a maximum value. Beyond this 
maximum, the output falls to zero due to "cathodal block". These relations are time-dependent. 
For example, the slope of the wave-pulse transfer function decreases with time when subjected 
to sustained activation ("adaptation") Q. 

The net transformation of a input by a neuron is therefore given by the combined action of 
the two transfer-functions. Let us approximate the nonlinear pulse-wave transfer function F\ 
with a piecewise linear function, such that 

Fi(z) = —c, if z < —c/m, 

= m z, if — c/m < z < 1/m, (4) 
= 1, if z > 1/m. 
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The wave-pulse transfer function F2 is represented as 



F 2 (z) = 0, if z< t, 

= m'(z-t), if t <z< t + (l/m'), 
= 0, if z> t + (l/m'), 



(5) 



where m,m' are the slopes of Fi,F% respectively, c is the inhibitory saturation value and t 
represents a threshold value. 

It is easily seen that the combined effect of the two gives rise to the resultant transfer 
function, G, defined as 



G(z) = 0, if z < t, 

= mm'(z-t), if t < z < t+(l/m), 

= m'(l-t), if t+(l/m) <z< t+(l/m'), 

= 0, if z > t+{l/m'). 



(6) 



In the present work we will assume that m' (1 — t) « t + (1/m'). This condition ensures that 
the operating region of the neuron does not go into the "cathodal block" zone. This allows us 
to work with the following simplified neural activation function (upon rescaling) throughout the 
rest of the paper: 

F a (z) = 0, if z< t, 

= a(z-t), if t < z < i+(l/a), (7) 
= 1, if z > t+(l/a), 

where a (> 0) is called the gain parameter of the function (Fig. 1). For infinite gain (a — > 00), 
the activation function reverts to the hard-limiting Heaviside step function. 
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Figure 1 The piecewise linear activation function F for a single neuron (gain parameter, a 
5). 
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Note that, if neural populations had been considered, instead of single neurons, then sig- 
moidal activation functions, e.g., 

F,(z) = 1/(1 + e-n, 

would have been the appropriate choice. By varying fj,, transfer functions with different slopes 
would be obtained. 

Although, in the present study, the gain parameter, a, of the transfer function is considered 
constant, in general it will be a time- varying function of the activation state, decreasing under 
constant external stimulation until the neuron goes into a quiescent state. The threshold t is 
also a dynamic parameter, changing as a result of external stimulation. We have also assumed 
that the neuron state at the nth instant is a function of the state value at the previous instant 
only. Introducing delay effects into the model, such that, 

X n +i = F{X n , X n _i, . . . ,X n - T ), 

might lead to novel behavior. 



3. Excitatory-Inhibitory Pair Dynamics 

Having established the response properties of single neurons, we can now study the dynam- 
ics when they are connected. It is observed that, even connecting only an excitatory and an 
inhibitory neuron with each other leads to a rich variety of behavior, including high period os- 
cillations and chaos. The continuous-time dynamics of pairwise connected excitatory-inhibitory 
neural populations (with sigmoidal nonlinearity) have been studied before However, in the 
present case, the resultant system is updated in discrete-time intervals and the dynamics is gov- 
erned by piecewise linear mappings. This makes chaotic behavior possible in the proposed neural 
network model. Chaotic activity has been previously observed in piecewise linear systems, for 



both continuous-time [11] as well as discrete-time evolution [12] of the system. 

If X and Y be the mean firing rates of the excitatory and inhibitory neurons, respectively, 
then their time evolution is given by the coupled difference equations: 

X n+1 = F a (W xx X n - W xy Y n ), (8) 

Y n+1 = F b (W yx X n - W yy Y n ). 



The network connections are shown in Fig. 2. Without loss of generality, the connection 
weightages W xx and W yx can be absorbed into the gain parameters a and b and the correspond- 
ingly rescaled remaining connection weightages, W xy and W yy , are labeled k and k' respectively. 
For convenience, a transformed set of variables, Z n = X n — k Y n and Z' n = X n — k! Y n , is used. 
The dynamics is now given by 



Z n+1 = F a (Z n ) - k F b (Z' n ), 



0) 
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► excitatory a Inhibitory 

Figure 2 The pair of excitatory (x) and inhibitory (y) neurons. The arrows and circles represent 
excitatory and inhibitory synapses, respectively. 

Z' n+1 = F a {Z n ) - k' F b (Z' n ). 

Note that, if k = k', the two-dimensional dynamics is reduced to an effective one-dimensional 
one, simplifying the analysis. We will now examine the cases: (i)k = k' = 1, {ii)k = k' ^ 1, and 
(iii)k ^ k', in detail. Unless mentioned otherwise, the threshold t will be taken as (a non-zero 
value of t introduces an affine transformation to F) . 

3.1. k = k' = 1 

This represents the condition when the connection weights W xy = W xx and W yy = W yx , (a > 6). 
The dynamics is that of an asymmetric tent map (Fig. 3(a)): 

Z n+ i = (a - b) Z n , if < Z n < 1/a, 

= 1 - bZ n , if 1/a < Z n < 1/6, (10) 
= 0, otherwise. 

The fixed points of this system are, Z* = and Z| = 1/(1 + b). Z* is stable for a — b < 1, 
whereas Z| exists only when a — b > 1, and is stable for b < 1. Beyond this, chaotic behavior is 
observed unless the maximum output value, i.e., 1 — (b/a), iterates to the region where Z > 1/6. 
The parameter space diagram is shown in Fig. 4. 

Along the line b/a = 0.5, we get the symmetric tent map scenario. So the Lyapunov exponent 
along this curve grows as A = log e (6) for < a < 4. This is one of the two special cases where 
an analytical expression for A can be obtained. The other instance is when the map's invariant 
probability distribution, P(Z) = 1. This occurs when 



F (1/a) = 1 - (b/a) = 1/6. 



(11) 
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Figure 4 The activation gain a vs. (b/a) parameter space for k = k' = 1. Region A: z* = 
stable, B: z* = 1/(1 + 6) stable, C: chaos, D: coexistence of z* = and a fractal chaotic invariant 
set. 
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Along the curve defined by the above relation, the Lyapunov exponent evolves with the param- 
eter b/a according to 



A = -b/a log e (6/o) - (1 - (6/o))log e (l - (b/a)). 



(12) 




Figure 5 Lyapunov exponent of the chaotic dynamics for k 
the entire interval [0, 1/6] is uniformly visited. 



k' = 1 and a = 4.0. At b/a = 0.5, 



In general, A has to be obtained computationally. Fig. 5 shows A plotted against b/a 
for a = 4, when the map is in the chaotic region. A sharp drop to zero is observed in both 
the terminal points, indicating sharp transition between chaotic and fixed-point behavior at 
b/a = 0.25 and 0.75. At b/a = 0.5, the entire interval [0, 1/6] is uniformly visited by the chaotic 
trajectory (P(Z) = 1). This corresponds to "fully-developed chaos" in the symmetric tent map 
for which A = log e (2) ~ 0.693. 

When F(l/a) > 1/6, the interval [0,1/6] is divided into a chaotic region of measure zero, 
defined on a non-uniform Cantor set (in general) and an "escape set" which maps to Z\ = 0. 
This is because, for Z £ (l/6(a — 6), (6 — l)/6 2 ), F(Z)=0. Any time an iterate of Z falls in this 
region, in the next iterate the trajectory will converge to Z\. The points left invariant after 
one iteration, will be in the two intervals [0, l/6(a — 6)] and [(6 — l)/6 2 , 1]. The phase space is 
thus fragmented into two invariant regions. After n iterations, there will be 2 n fragments of the 
chaotic invariant set, with n\/r\(n — r)\ (r = 0,1, ... ,n) intervals of length (a — 6) r (l — 6) r ~ n . 
The fragmentation of the phase space, therefore, has a multifractal nature [fQjfl . 

The presence of multiple length scales is due to the fact that the slope magnitude of the map 
is not constant throughout the interval [0, 1/6]. It is to be noted that, even for Z not belonging to 
the fractal invariant set, the trajectory might show long chaotic transients until at some iterate 
it maps to Z* = 0. For b/a = 0.5, the map has a constant slope. As a result, the Cantor set is 
uniform, having exact geometrical self-similarity and a fractal dimension, D = log e (2)/log e (6). 
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So, the phase space of the coupled system has a fractal structure in this parameter region, i.e., 
where 1 — (b/a) > 1/6. 




Figure 6. Bifurcation diagram for k = k' = 1 at a = 4.0. 



Fig. 6 shows the bifurcation structure of the map for a = 4. For b/a < 0.25, the fixed point 
Z£ is stable. At b/a = 0.25 it becomes unstable, leading to bands of chaotic behavior. The 
chaotic bands collide with the unstable fixed point Z% at b/a ~ 0.2985... and merge into a single 



chaotic band. This band-merging transition is an example of crisis [14] and has been studied 
in detail for the symmetric tent map [J15[| . The b- value at which the band-merging occurs for a 
given value of a, can be obtained analytically by solving the quartic equation: 



6 4 + (1 - 2a)b 3 + (a 2 - a)b 2 + ab+(a 



0. 



(13) 



For 2 < a < 2.5, all the roots are complex, implying that band-merging does not occur over this 
range of a-values. 

Uniform chaotic behavior occurs at b/a = 0.5 (the entire interval [0,1/b] is uniformly visited 
by the chaotic trajectory). The chaotic band collides with the unstable Z| again at b/a = 0.75. 
This boundary crisis destroys chaos and stabilizes the fixed point Z\ = 0. 
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3.2. jfe = k' ^ 1 

This represents the condition when the connection weightages are such that, W xy /W xx = 
Wyy/Wy X = k, (a > 6) . The dynamics is given by the following map (Fig. 3(b)) 

Z n+1 = (a - kb) Z n , if < Z n < 1/a, 

= 1 - kbZ n , if 1/a < Z n < 1/6, (14) 
= 1 — k, otherwise. 

The key difference with the earlier case is that now the dynamics supports superstable period-m 
orbits (m > 2). This is a result of the existence of a region of zero slope [Z > 1/6) giving a 
non-zero output. There are two fixed points of the map, Z\ = (as before), and, 

Zl = 1 - jfe, if < k < 1 - (1/6), or, 

= 1/(1 + kb), if (a - l)/6 > k > 1 - (1/6). 

Z% = l — k, if it exists, is superstable, as the local slope is zero. On the other hand, Z\ = 1/(1 + 
kb) is stable, only if bk < 1. If the fixed points are unstable, but iterates of Z fall in the region 
Z > 1/6, superstable periodic cycles occur. The fixed point, Z\ = 0, becomes stable when 
(a — bk) < 1. Chaotic behavior occurs if none of the fixed points are stable, and no iterate of 
Z falls in the region Z > 1/6. The (b/a) vs. k parameter space diagram in Fig. 7 (for a = 4) 
shows the different dynamical regimes that are observed. 




b/a 

Figure 7 The (b/a) vs. k parameter space for k = k' ^ 1 at a = 4.0. Region A: z* = 1/(1 + kb) 
stable, B: z* = 1 — k stable, C: superstable periodic cycles, D: chaos, E: z* = stable. 

The bifurcation diagram for a = 4, 6 = 2 (Fig. 8) shows how the dynamics changes with k. 
For < k < 0.5, Z| = 1 — k, is the stable fixed point. At k = 0.5, Z| becomes unstable, giving 
rise to a superstable period-2 cycle. A periodic regime is now observed, which was absent in the 
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Figure 8. Bifurcation diagram for k = k' 7^ 1 at a = 4.0, b = 2.0. 



previous case. The periodic orbits initially follow a period- doubling sequence until a period-32 
(= 2 x 2 4 ) orbit gives rise to a period-48 (= 3 x 2 4 ) one. This occurs as a result of border-collision 
bifurcations by which "period-2 to period-3" bifurcations have been seen to occur [12]. In the 
above instance, each of the sixteen period-2 orbits give rise to a period-3 orbit. The structure of 
the superstable periodic orbits is quite complex. The length of the cycles is plotted against k in 
Fig. 9. The remarkable self-similar structure of the intervals is to be noted. Numerical studies 
indicate that cycles of all periods exist having the following ordering: between any superstable 
period-m and period-(m + 1) cycle, there exists an interval of k for which a period-(m + 2) orbit 
is superstable. At k = 1.0 all periodic orbits become unstable, leading to onset of chaos. The 
chaotic behavior persists till k = 1.5, when Z* = becomes stable. 

Let us now study the effect of introducing a threshold. A positive threshold, to > 0, applied to 
both F a and Ft, has the effect of shifting the map to the right by to- This allows the segmentation 
of activation state (X, y)-space, according to dynamical behavior. For initial conditions lying 
in the region bounded by the two straight lines, Y = (X — to)/k and Y = (X — to)/(k — 1/ak), 
the trajectories are chaotic, provided the maximum point of the map, F(Z) = 1 — (kb/a), does 
not iterate into the region Z > to + 0-/b)- For the region, Y > (X — to)/k, any iterate will map 
to the fixed point, Z* = 0. Initial conditions from Y < {X — to)/(k — 1/ak) will map to the 
chaotic region, if the maximum point of the map does not iterate into Z > to + (1/6). Otherwise, 
a fractal set of initial conditions will give rise to bounded chaotic motion, the remaining region 
falling in the "escape set" , eventually leading to periodic orbits. 

A negative threshold shifts the map to the left. This has the consequence of destabilizing 
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Figure 9 Length of superstable periodic cycles, m, of the excitatory-inhibitory neural pair 
(a = 4, 6 = 2) for (a) 0.75 < k < 1, and (b) 0.82 < k < 0.84. Note the self-similar structure of 
the intervals. 



the fixed point Z* = 0. If the other fixed point is also unstable then this will imply that all 
possible (X, Y) values will give rise to periodic or chaotic trajectories. This indicates the possible 
existence of a global chaotic attractor. 

3.3. k^k' 

This corresponds to the condition when all the connection weights are different. The dynamics 
is irreducible to 1-dimension. We need to consider only the positive (Z, Z') region, as otherwise, 
(0, 0) is the stable fixed point. In the non-zero region, different dynamical behavior may occur 
depending on the region where the fixed point occurs and on its stability. One of the fixed 
points is (Z,Z r ) = (0,0), whose stability is determined by obtaining the eigenvalues of the 
corresponding Jacobian, 

_ a —kb 
J = a -k'b 

Evaluating the above matrix, gives the following condition 

- 2 < (a - k'b) ± [(a - k'b) 2 - 4ab(k - k')] 1 ' 2 < 2, (15) 

for stability of the fixed point. 

The other fixed point may occur in any one of the four following regions of the (Z, Z')-space: 

Region I: < Z < l/o, < Z' < 1/6. 
(Z, Z') = (0, 0) is the only fixed point. 

Region II: < Z < 1/a, Z' > 1/6. 
The fixed point is (Z, Z') = (k/(a - 1), (a(k - k') + k')/(a - 1)), which is stable if -1 < a < 1. 
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Region III: Z > 1/a, < Z' < 1/6. 
The fixed point is (Z,Z') = ((l + b(k' -k))/(l + bk'),l/(l + bk')), which is stable if — 1 < k'b < 1. 

Region IV: Z > 1/a, Z' > 1/6. 
The fixed point is (Z, Z') = (1 — k, 1 — k'). This is a superstable root, as the local slope is zero 
under all conditions. 

The abundance of tunable parameters in this case makes detailed simulation study extremely 
difficult. However, some preliminary studies in the (k,k') parameter space (keeping the other 
parameters fixed) gives indication of dynamics similar to that seen in cases (i) and (ii). The 
(k, k') parameter space is shown in Fig. [l0| for a = 4, b = 2. 




k 



Figure 10 The (k, k') parameter space for a = 4.0 and b = 2.0. Region 1: x*=l, y*=l stable, 2: 
x*=l, y has period-2 cycles, 3: x*=0, y*=0 stable, a: Both x and y have period-2 cycles, b: x 
and y show period-m cycles (m > 2). Fully chaotic behavior occurs in the dark wedge-shaped 
region in 3. In addition, fractal intervals showing chaos occur in region b. 

A variety of dynamical behaviors is observed - from fixed points to periodic cycles to chaos, 
as indicated by the different regions. In addition, there are regions exhibiting periodic behavior 
which have fractal intervals of chaotic activity embedded within them. 

4. Extension to Large Networks 

In the previous section, the behavior of a pair of excitatory-inhibitory neurons (number of 
neurons, N = 2) was shown to have sufficient complexity. The dynamics of a N-neuron network 
(N » 2) described by 

X n+1 = F(]TW.X n ), 

where X n is the set of N activation state values (both excitatory and inhibitory neurons), and 
W is the matrix of connection weights between different neurons. The full range of behavior 
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shown by such a system will be impossible to study in detail, as the number of available tunable 
parameters are too large to handle. However, under certain restrictions, the dynamics of such 
large networks can be inferred. 

Let Wij denote the connection weight from jth to the ith neuron. Then, under the condition 

Wi j + i/Wi i = kj, (kj = constant for a given j = l,...,N), (16) 

the N-neuron network dynamics is reducible to that of a 1-dimensional map with (N + 1) linear 
segments (for t = 0). The occurrence of "folds" in a map have already been shown to be 
responsible for creation and persistence of localized coherent structures within a chaotic flow 
PI . As in this case, the resultant map will have a number of such folds, the system might 
show coexistence of multiple chaotic attractors (isolated from each other). A simple example 
to illustrate this point is a fully connected network of four neurons: two excitatory (x\,X2) and 
two inhibitory (2/1,2/2)- Let dj and bi represent the slope of the transfer functions for the ith 
excitatory and inhibitory neurons, respectively. The 4-dimensional dynamics is reducible to the 
1-dimensional dynamics of z = x\ — k\y\ + — ^32/2- Simulations were carried out for the set 
of parameter values: {a\ = 4.6, 02 = 4.0, 61 = 3.6,^2 = 1-6) and [k\ = 0.7, &2 = 1-0,^3 = 1.1). 
Furthermore, £2,2/2 have a threshold equal to l/b\. Fig. 11(a) shows the return map and time 
evolution of z in the absence of any bias. There is only a single global chaotic attractor in 
this case. When a small negative bias is applied to the whole network, the previous attractor 
splits into two coexisting isolated attractors having localized chaotic activity. Which attractor 
the system will be in, depends upon the initial value it starts from. Fig. 11(b) shows the 
return map for a bias value of -0.15 and the superposed time evolutions of z starting from initial 
conditions belonging to two different attractors. So, an increase in bias, can cause transition 
from global chaos to localized chaotic regions. 

This property can be used to simulate a proposed mechanism of olfactory information pro- 
cessing It has been suggested that the olfactory system maintains a global attractor with 
multiple "wings", each corresponding to a specific class of odorant. During each inhalation, 
the system moves from the central chaotic repeller to one of the wings, if the input contains a 
known stimulus. The continual shift from one wing to another via the central repelling zone has 
been termed as chaotic "itinerancy" . This forms the basis of several chaotic associative memory 
models. 

The above picture can be observed in the present model by noting that, if the external input 
has the effect of momentarily increasing the bias from a negative value to zero, then the isolated 
chaotic regions merge together into a single global attractor. In this condition, the entire region 
is accessible to any input state. However, as the bias goes back to a small negative value, the 
different isolated chaotic attractors re-emerge, and the system dynamics is constrained into one 
of these. Sustained external stimuli will cause the gain parameters to decrease (adaptation), 
thereby decreasing the local slope of the map. If the stimulus is maintained, the unstable fixed 
point in the isolated region will become stable leading to a fixed-point or periodic behavior. 
The above scenario, in fact, is the basis of using the proposed model as an associative memory 
network. 
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Figure 11 The return map and time evolution of the reduced variable, z, for a 4-neuron network 
(for details see text), with (a) bias = and (b) bias = - 0.15. In the former there is a single global 
chaotic attractor. For non-zero bias, there are two co-existing chaotic attractors. Time evolution 
of z starting from two initial conditions belonging to different attractors are superposed. 
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5. Information Processing with Chaos 

Chaotic dynamics enables the microscopic sensory input received by the brain to control the 
macroscopic activity that constitutes its output. This occurs as a result of the selective sensitiv- 
ity of chaotic systems to small fluctuations in the environment and their capacity for rapid state 
transitions. On the other hand, chaotic attractors are globally extremely robust. These prop- 
erties indicate that the utilization of chaos by biological systems for information processing can 
indeed be advantageous. It has been suggested, based on investigations into cellular automata, 



that complex computational capabilities emerge at the "edge of chaos" [17]. 

Based on this notion, efforts are on to use chaos in neural network models to achieve human- 
like information processing capabilities. Chaotic neural networks have been already been applied 
in designing associative memory networks || and solving combinatorial optimization problems, 
using chaos to carry out an effective stochastic search Q. The superposition of chaotic maps 
for information processing has also been suggested before p| . 

The model presented in this paper can be used for a variety of purposes, classified as follows: 

Associative memory: A set of patterns (i.e., specific network state configurations) are 
stored in the network as attractors of the system dynamics, such that, whenever a distorted 
version of one of the patterns is presented to the network as input, the original is retrieved upon 
iteration. The distortion has to be small enough so that the input pattern is not outside the 
basin of attraction of the desired attr actor. In networks using convergent dynamics, the stored 
patterns necessarily have to be time-invariant or at most, periodic. 

Chaos provides rapid and unbiased access to all attractors, any of which may be selected 
on presentation of a stimulus, depending upon the network state and external environment. It 
also acts as a "novelty detector", classifying a stimulus as being previously unknown, by not 
converging to any of the existing attractors. This suggests the use of chaotic networks for auto 
association. 

In the previous section, the basic mechanism for constructing an associative memory network 
has been described. In this proposed model, both constant and periodic sequences can be 
stored. This is made possible by introducing "folds" in the return map of the network, so that 
a large number of isolated regions are produced. The nature of the dynamics in a region can 
be controlled by altering the gain parameters of individual neurons. Accessibility to a given 
attractor depends upon the initial condition of the network and the input stimulus. So, regions 
with fixed-point or periodic attractors may be embedded within regions having chaotic behavior. 
In addition, chaotic trajectories confined within a specific region can also be generated when 
presented with a short-duration input stimulus belonging to that region. "Novelty detection" is 
implemented in the above model by making the basins of attractors (corresponding to the stored 
patterns) of some pre-specified size. Input belonging outside the region, therefore, cannot enter 
the basin and will not be able to converge to the stored pattern. 

Pattern classification: In this information processing task, different input sets need to be 
classified into a fixed number of categories. Decision boundaries, i.e., boundaries between the 
different classes are constructed by a "training session" where the network is presented with 
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a series of inputs and the corresponding class to which they belong. In the proposed model, 
classification can be on the basis of dynamical behavior. For example, input sets belonging to 
different input classes may give rise to different periodic sequences. Otherwise, the distinction 
can be made between categories of inputs which give rise to chaotic and non-chaotic trajectories. 
For a pair of neurons (N = 2), under the condition k = k', linear separation of the (X, y)-space 
can be done (as shown above). By varying the parameters k and b, the orientation and size 
of the class regions can be controlled. If k ^ k' and N > 2, nonlinear decision boundaries 
between different classes can be generated. By using suitably adjusted weights, any arbitrary 
classification can be achieved. 




Figure 12 (a) A fully connected 4-neuron network with 2 excitatory (xi 2) an d 2 inhibitory (2/1,2) 
neurons. The arrows and circles represent excitatory and inhibitory synapses, respectively, (b) 
The (xi,X2) phase space shows the basin of the chaotic attractor (shaded region) for threshold, 
i=0.25. The unshaded region corresponds to fixed point behavior of the network. 



An example of nonlinear decision boundary generation is shown in Fig. 12. The network 
used for this purpose consists of 4 neurons - 2 excitatory (xi,X2) and 2 inhibitory (2/1,2/2) (fig. 
12(a)). The gain parameters are a xltX2 = 2 and b yit y 2 = 1. All the neurons have a threshold, t. 
The network is fully connected with all weightages equal to unity. The input stimulus is taken 
to be the initial value of the excitatory neurons and the inhibitory neuronal states are initially 
taken to be zero. 

As shown in fig. 12(b), for t = 0.25, the (xi, X2) phase space is segmented into basins leading 
to either fixed point or chaotic attractors. By increasing t, the width of the chaotic band can 
be reduced. Changing the initial value of the inhibitory neurons will cause translation of the 
band and manipulating the connection weights gives a rotation to the band. Thus, any general 
transformation can be applied to the segment. More complex network connections might permit 
segmenting isolated box-like regions in the phase space. This possibility is currently under 
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investigation. 

System Dynamics Approximation: A system may be described by a nonlinear input- 
output relation, 

Y = G(X), 

where the mapping function, G, is unknown. By having access to a limited set of input-output 
pairs, the function has to be approximated - in effect, building a system simulator. In the present 
model, a sufficient number of coupled neurons can be used to construct any arbitrary piecewise 
linear input-output relation. By use of a suitable learning rule, the available data set can be 
used to determine the gain parameters, thresholds and connection weights of the network. A 
close approximation of the system dynamics will enable prediction and control of its behavior. 
The approximation's accuracy is not restricted to systems with piecewise linear functions - but 
can also give good qualitative reconstruction of smooth nonlinear systems. 

Periodic sequence generation: Capability for periodic sequence generation can be ex- 
ploited for modeling central pattern generators. These are a class of biological neural ensembles 
which control well-defined rhythmic muscle movements such as swimming, running, walking, 
breathing, etc. Usually they are found in the spinal cord, producing periodic sequences without 
feedback from the motor system or higher-level control. The ability to generate multiple se- 
quences from the same neural assembly is another interesting feature. Postulating the existence 
of single pacemaker neurons acting as the 'system clock' to initiate periodic activity cannot 
explain all the observed phenomena. The existing network models for simulating this behavior 
mostly suffer from the drawback that they cannot generate multiple non-overlapping sequences. 
This shortcoming is overcome in the model presented here. For N = 2 and k = k' 7^ 1, a rich 
variety of periodic sequences can be chosen from the same network, simply by altering the gain 
parameters by a very small amount. As mentioned above, numerical investigations indicate that 
cycles of any period can be generated by suitably altering the value of k. 

6. Discussion and Future Work 

The network has been studied in detail only for N = 2. Nonetheless it shows capability for 
supporting extremely complex behavior. Under certain restrictions, the dynamics for N >> 2 
networks can also be understood. Relaxation of these restrictions will provide a challenging task 
for the future. 

One important point not touched here is the issue of learning. The connection weights {Wjj} 
have been assumed constant, as they change at a much slower time scale compared to that of 
the activation state values. However, modification of the weights due to learning will also cause 
changes in the dynamics. Such bifurcation behavior, induced by weight changes, will have to be 
taken into account when devising learning rules for specific purposes. The interaction of chaotic 
activation dynamics at a fast time scale and learning dynamics on a slower time scale might 
yield richer behavior than that seen in the present model. An interesting study for the future 
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will be to introduce time-varying Hebbian synaptic weights, evolving according to 

Wij(n + 1) = Wij{n) + eX t {n)Xj(n), 

where X(n) and W(n) denote the neuron state and connection weight at the nth instant, and 
e is related to the time-scale of the synaptic dynamics. The combined effect of such synap- 
tic dynamics with the chaotic network dynamics (for small N) described above might lead to 
significant departure from the overall features described here. 

Real biological systems reside in an extremely noisy environment. This is incorporated 
into neural models by using stochastic updating rule and/or explicit introduction of a term 
representing external noise. We plan to introduce similar features in the proposed model. In 
dissipative chaotic systems, the effect of external noise seems to be limited to destroying the fine 



structure of the bifurcation sequence IS]. The interaction of deterministic chaos and stochastic 
noise in the network will be interesting to study. 

Another possible modification to the present model will be to incorporate time- varying oper- 
ating conditions. In || time-dependence of a suitable system parameter was shown to give rise 
to interesting dynamical behaviors, e.g., transition between periodic oscillations and chaos. This 
suggests that varying the environment can facilitate memory retrieval if dynamic states are used 
for storing information in a neural network. In the proposed network, periodic variations can 
be incorporated into the available system parameters: gain, threshold and connection weights. 
In nonlinear systems, time- varying dynamics often leads to the phenomenon of "stochastic res- 
onance" where a subthreshold signal is amplified by the background noise. It has been observed 
that replacing noise by chaos also gives rise to this resonance behavior in a bimodal piecewise 



linear map [20]. It will be of interest to see whether such phenomena can occur in the present 
model, on the introduction of temporal variation in the parameters. The occurrence of "res- 
onance" might enhance the capability of the network to detect subthreshold signals, through 
signal amplification by deterministic 'noise'. 
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